Named Test Suite

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 592 of xUnit Test Patterns for the latest information.
How do we run the tests when we have arbitrary groups of tests to run?

Define a test suite, suitably named, that contains a set of tests that we wish to be able to run as a group.

Sketch Test Suite Factory embedded from Test Suite Factory.gif

When we have a large number of tests, we need to organize them in a systematic way. This allows us to group tests for related functionality close to each other. We want to be able to run all the tests for the entire application or component easily but we also want to be able to run all the tests for specific subsets of the functionality or subcomponents of the system. There are also many situations in which we want to run only a subset of all the tests we have defined.

Named Test Suites give us a way to choose which pre-defined subset of the tests we want to run.

How It Works

For each group of related tests that we would like to be able to run as a group, we can define a special Test Suite Factory (see Test Enumeration on page X) with an intent-revealing name. The Factory Method[GOF] can use one of several test suite construction techniques to return a Test Suite Object (page X) containing only the specific Testcase Objects (page X) we wish to execute.

When To Use It

We often want to run all the tests with a single command but there are many reasons for wanting to run a subset of tests. The most common one is time and for this purpose, running the AllTests Suite for a specific context is probably our best bet. When the tests we want to run are scattered across multiple contexts and some contexts contain tests we definitely don't want run, we can use a Subset Suite.

Variation: AllTests Suite

We often want to run all the tests we have. With smaller systems we often make it standard practice to run the AllTests Suite after checking out a new code base (to make sure we are starting at a known point) and before every check-in (to ensure all our code works.) We typically have an AllTests Suite for each package or name space of software so that we can run subsets of the tests after each code change as part of the "Red, Green, Refactor" cycle.

Variation: Subset Suite

A common reason for not wanting to run tests is because they are Slow Tests (page X). Tests that exercise components that access a database are going to be much slower than tests that run entirely in memory. By defining a Named Test Suite for the database tests and another for the in-memory tests, we can choose not to run the database tests by merely running the in-memory Subset Suite.

Another common reason for not running tests is because the context they need to run is not available. For example, if we don't have a web-server running on our development desktop, or deploying our software to the web server takes too long, we won't want to run the tests of components that require the web-server to be running (they would just take extra time to run and we know they wil fail and spoil our chances for a green bar.)

Variation: Single Test Suite

The degenerate for of a Subset Suite is the Single Test Suite in which we instantiate a single Testcase Object so that we can run a single Test Method (page X). This is particularly useful when we don't have a Test Tree Explorer (see Test Runner on page X) available or when the Test Method requires some form of Setup Decorator (page X) to run properly. Some test automaters keep a "MyTest" Testcase Class (page X) open in their workspace at all times specifically for this purpose.

Implementation Notes

The concept of running named sets of tests is independent of how we build the Named Test Suites. For example, we can use Test Enumeration to build up our suites of tests explicitly or we can use Test Discovery (page X) to find all the tests in a particular place (e.g. name space or assembly). We can also do Test Selection (page X) from within a suite of tests to create a smaller suite dynamically. Some members of the xUnit family require us to define the AllTests Suites for each test package or subsystem manually while others, like NUnit automatically create a Test Suite Object for each namespace.

When we are using Test Enumeration and we have Named Test Suites for various subsets of the tests, it is better to define our "AllTests Suite" in terms of these subsets. By implementing AllTests as a Suite of Suites (see Test Suite Object) we will only ever have to add a new Testcase Class to a single Named Test Suite which will then be rolled up into the AllTests Suite for the local context as well as the Named Test Suite and the next higher context.

Refactoring Notes

The steps to refactor to Named Test Suite are highly dependent on the variant of Named Test Suite we are using so I'll dispense with the motivating example and skip directly to the various examples of Named Test Suite.

Example: AllTests Suite

An AllTests Suite helps us run all the tests for different subsets of the functionality of our choosing. For each subcomponent or context (e.g. a Java package), we define a special test suite (and its corresponding Test Suite Factory) called "AllTests". In the suite Factory Method on the Test Suite Factory we add all the tests in the current context and all the Named Test Suites from any nested contexts (such as nested Java packages). That way, when the top level Named Test Suite is run, all the Named Test Suites for the nested contexts will also be run.

The following illustrates the kind of code that would be required to run all the tests in most members of the xUnit family:

public class AllTests {

   public static Test suite() {
      TestSuite suite = new TestSuite("Test for allJunitTests");
      //$JUnit-BEGIN$
      suite.addTestSuite( com.clrstream.camug.example.test.InvoiceTest.class);
      suite.addTest(com.clrstream.ex7.test.AllTests.suite());
      suite.addTest(com.clrstream.ex8.test.AllTests.suite());
      suite.addTestSuite( com.xunitpatterns.guardassertion.Example.class);
      //$JUnit-END$
      return suite;
   }
}
Example AllTestsGenerated embedded from java/allJunitTests/AllTests.java

We've had to use a mix of methods because we are adding other Named Test Suites as well Test Suite Objects representing a single Testcase Class. In JUnit we use different methods to do this but other members of the xUnit family may use the same method signature.

The other thing to note here is the JUnit-start and JUnit-end comments. The IDE (in this case Eclipse) is helping us by automatically regenerating the list between these two comments. This is a semi-automated form of Test Discovery.

Example: Special Purpose Suite

Suppose we have 3 major packages (A,B and C) containing business logic and each contains both in-memory objects and database access classes. We would then have corresponding test packages for each of these 3 packages and some tests in each package will require the database while others can run purely in memory.

We want to be able to run:

All the tests
All the database tests
All the in-memory tests

for the entire system, and for each package A, B and C. This implies a total of 12 named sets of tests (3 named sets for each of 4 contexts.)

In each of the 3 packages A, B and C, we should define the following Named Test Suites:

AllDbTests, by adding all the Testcase Classes containing database tests.
AllInMemoryTests, by adding all the Testcase Classes containing in-memory tests.
AllTests, by composing AllDbTests and AllInMemoryTests.

Then, at the top level testing context, we define Named Test Suites by the same names as follows:

AllDbTests, by composing all the AllDbTests Testcase Classes from packages A, B and C.
AllInMemoryTests, by composing all the AllInMemoryTests Testcase Classes from packages A, B and C.
AllTests, by composing all the AllTests Testcase Classes from packages A, B and C. (This is just the normal AllTests Suite.)

If we find ourselves needing to include some tests from a single Testcase Class in both Named Test Suites, we should split the class into one class for each context (e.g. database and in-memory).

Example: Single Test Suite

In some circumstances, especially when we are using a debugger, it is highly desirable to not run all the tests in a Testcase Class. One way to do this is to use the Test Tree Explorer provided by some Graphical Test Runners (see Test Runner). But when this capability isn't available, a common practice is to disable the tests we don't want run by either commenting them out, copying the entire Testcase Class and deleting most of the tests, or changing the names or attributes of the test that cause them to be included by the Test Discovery algorithm.

public class LostTests extends TestCase {
   public LostTests(String name) {
      super(name);
   }
  
   public void xtestOne() throws Exception {
      fail("test not implemented");
   }
  
   /*  
   public void testTwo() throws Exception { fail("test not implemented");
   } */
  
   public void testSeventeen() throws Exception {
      assertTrue(true);
   }
}
Example DisabledTestMethods embedded from java/com/xunitpatterns/mytest/LostTests.java

All of these approaches suffer from a possibility of Lost Tests (see Production Bugs on page X) if the means of running a single test is not reversed properly when the need has passed. A Single Test Suite makes it possible to run the specific test(s) without making any changes to the Testcase Class in question. This technique takes advantage of the fact that most implementations of XUnit require a one-argument constructor on our Testcase Class; this argument is the name of the method that this instance of the class will invoke using reflection. The one-argument constructor gets called once for each Test Method on the class and the resulting Testcase Object is added to the Test Suite Object. (This is an example of the Pluggable Behavior[SBPP] pattern.)

We can cause a single test to be run by implementing a Test Suite Factory class with the single method suite which creates an instance of the desired Testcase Class by calling the one-argument constructor with the name of the one Test Method to be run. By returning a Test Suite Object containing only this one Testcase Object from suite , we achieve the desired result (running a single test) without touching the target Testcase Class.

public class MyTest extends TestCase {
   
   public  static Test suite() {
      return new LostTests("testSeventeen");
   }
}
Example MyTest embedded from java/com/xunitpatterns/mytest/MyTest.java

I like to keep a Single Test Suite class around all the time and just plug in whatever test I want to run by changing the import statements and the suite method. Often, I'll have several Single Test Suite classes so I can flip back and forth between different tests very quickly. I find this easier to do than drilling down in the Test Tree Explorer and picking the specific test to run manually. (Your mileage may vary!)

Example: Smoke Test Suite

We can take the idea of a Special Purpose Suite and combine it with the implementation technique of a Single Test Suite to create a Smoke Test[SCM] suite. This involves picking a representative test or two from each of the major areas of the system and including them in a single Test Suite Object.

public class SmokeTestSuite extends TestCase {
   public  static Test suite() {
      TestSuite mySuite = new TestSuite("Smoke Tests");
      mySuite.addTest( new LostTests("testSeventeen") );
      mySuite.addTest( new SampleTests("testOne")     );
      mySuite.addTest( new FlightManagementFacadeTest(
         "testGetFlightsByOriginAirports_TwoOutboundFlights"));
      // add additional tests here as needed ...return mySuite;
   }
}
Example SmokeTestSuite embedded from java/com/xunitpatterns/mytest/SmokeTestSuite.java

This won't test our system thoroughly but it is a quick way to find out that some part of the core functionality is broken.

Page generated at Wed Feb 09 16:39:44 +1100 2011